Skip to content

[Subscription] Index consensus ACK progress by writer - #18835

Merged
jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/consensus-subscription-ack-scan
Oct 14, 2026
Merged

jt2594838 merged 1 commit into
apache:masterfrom
Caideyipi:fix/consensus-subscription-ack-scan

Conversation

@Caideyipi

@Caideyipi Caideyipi commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Description

ConsensusSubscriptionCommitState.stageCommittedAndAdvance previously found the next committed and outstanding entries by scanning the full backlog maps. With N outstanding entries from one writer and reverse-order ACKs, every ACK repeatedly scanned O(N) entries, producing a real O(N^2) ACK path.

I reproduced the behavior by calling the production ConsensusSubscriptionCommitState with a single writer:

backlog baseline reverse-ACK time indexed time
2,000 39.749 ms 3.116 ms
4,000 152.420 ms 5.222 ms
8,000 1,116.485 ms 5.737 ms

Each value is the best of three warmed runs. The baseline grows super-linearly as the backlog doubles; the indexed path does not. The new per-writer TreeSet indexes preserve the existing slot maps for exact ACK removal and make successor lookup logarithmic.

Remote-change check:

Testing

  • mvn -Ddevelocity.off=true spotless:apply -pl iotdb-core/datanode
  • mvn -Ddevelocity.off=true -pl iotdb-core/datanode checkstyle:check -DskipTests
  • ConsensusSubscriptionCommitStateTest: 19/19 passed, including a 2,048-entry reverse-ACK regression case.
  • git diff --check

The normal full DataNode compile is currently blocked by existing workspace SNAPSHOT/source inconsistencies (SubscriptionQueueAdmission and related classes) and a Windows compiler native-memory failure; the changed manager and targeted test were compiled directly against the existing DataNode build artifacts.

Changed files

  • ConsensusSubscriptionCommitManager
  • ConsensusSubscriptionCommitStateTest

@jt2594838
jt2594838 merged commit a7e508c into apache:master Oct 14, 2026
38 of 39 checks passed
@jt2594838
jt2594838 deleted the fix/consensus-subscription-ack-scan branch October 14, 2026 03:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants